Optional Stopping

Introduction Suppose you are determined to “prove” that green apples cause cancer. An Optional Stopping strategy (OS) is where you keep looking sampling experimental data until the observed correlation between eating green applies and cancer is significantly different from 0 (where “significantly” means that the null hypothesis is rejected by standard statistical tests). That is, you follow a rule that says “Don’t stop until you reject the null hypothesis”. This is also the best strategy for confirming the existence of UFOs or establishing the phenomenon of extrasensory perception (ESP) (see Feller, 1940, for the confutation of this tongue-in-cheek assertion). If the data are ‘noisy’ (and whose data are not?), then this will probably always work in principle, so not always in practice because you won’t live long enough to collect enough data. There are two schools of thought about optional stopping examples of the kind that I consider (see Robbins, 1952, for a more general discussion). The classical hypothesis testers say that it is a bad strategy if the probability of falsely rejecting the null hypothesis when it is true is 1. Some Bayesians, and likelihood theorists, say that it all that matters is how well the hypotheses fit the data, and it makes no difference whether you collect n data by an OS strategy, or if you collect n data with the prior intention of stopping at a sample size of n (strategy FS). About the only thing that has never been said about OS is that it is better than FS (with the same n). What follows is a series of computer simulation comparing an OS strategy with an FS strategy. The first simulation assumes that the null hypothesis is true, so that rejection of the null hypothesis is always a mistake. When I initially ran the simulation, I found that I had to wait too long for my computer finish doing its experiments (despite its running at 450 MHz). So, I set an upper limit of 2000 data points. For me, 2000 without stopping counts as “no outcome”. As you will see from the results below, the experiment had “no outcome” 40% of the time. The next two simulations show how things change when the null hypothesis is false. The results were initially surprising to me. It turns out that OS can be a more reliable method than FS most of time. In the final section, I explain this odd result in terms of an easy-to-understand analogy.